Workflow tools

Airflow, published by Apache. It has been tested by colleagues.

It requires an executor to launch the workers, such as celery or kubernetes.

Workflows (here called pipelines) are defined in Python without UI.

But it is possible to view workflows in progress via a UI.

A chart helm is also provided.

Quite a few generic job implementations are available via plugins: sending an e-mail, making an SFTP request, etc.

Full Python ecosystem.

Conductor from Netflix, used internally.

I've never used it and I don't have any feedback, so I'm relying on their documentation.

Workflows can be defined in JSON or via a UI. A debugging interface is provided, with time spent per job and precise control of each workflow.

Resilient and scalable.

Worker possible in Java, NodeJS, Python or C#.

[EDIT: project archived on 12/13/2023]

Fairly recent orchestration tool, written in Go with SDKs in various technologies (Python, Go, TS, PHP, Java).

There's a choice between self-hosted or a cloud service for the runtime server.

In terms of operation, it's pretty standard, with a call to launch the workflow, and workers that pull and execute a task when it's waiting, with options for retry, timeout, and so on.

Simple to deploy and robust, it runs well in production according to the feedback I've had, even on long workflows (= dozens of days).

Director of OVH.

An overlay at Celery.

Workflows are defined in YAML and simply call celery tasks.

A visualization UI is available.

It looks super simple and light. In a Python ecosystem where you're already doing celery, it's very easy to integrate.

Its great strength lies in its set of connection plugins, which can be interfaced with a large number of integrated services.
New tasks can be coded in JS or Python.

Similar to the younger n8n, but with a different approach: rather than drag&drop, workflows are defined in YAML.

The monitoring interface is clear, showing logs as well as the duration and status of each task.

Customized workers can be written in Python, R, Node.js, Julia, Ruby, Shell or PowerShell.

Close to Airflow's DAG approach

Zeebe, published by Camunda. I used it to manage order workflows for an e-commerce application, before it was replaced by Temporal.

Zeebe itself is opensource, but not the other tools in its ecosystem, which are difficult to do without in production and prohibitively expensive to license (€35,000/year for the right to deploy them in self-hosted production).

It's a pretty big thing that gave me a bad dev experience and the default prod configuration isn't robust (after all, I have no expertise in Java administration, so that can't help).

Retry automatic, workers are independent and can be written in any language via a GRPC communication system, but you need to think about implementing task idempotency yourself.

Workflows are defined by BPMN schema, which is handy for presenting to business teams, but in my experience, it's still the devs who modify the schema, so the gain is limited.

Comes with a second tool, Operate (paid license if used in production), for monitoring and graphically interacting with workflows in progress.

SaaS offer available for all this, we opted for self-hosting on our Kubernetes cluster, using the chart helm provided. The zeebe cluster cannot be dynamically resized (node added or removed) on the fly.

Nor is it possible to go backwards in a workflow, just stop it or restart a failed job.

The latest and yet one of the oldest: Uber's Cadence solution has just been upgraded to V1.

In preparation for 6 years, Temporal was originally a fork of this one.

It's all done in Go. Workflows are defined by code (Java or Go officially, Python or Ruby via community ports).

It has a UI for observability.

You can choose your data persistence solution, and PostgreSQL in particular is supported.